Feature Selection and Multi-Class Classification Using a Rule Ensemble Method

Ensemble methods for supervised machine learning have become popular due to their ability to accurately predict class labels with groups of simple, lightweight “base learners.” While ensembles offer computationally efficient models that have good predictive capability, they tend to be large and offer little insight into the patterns or structure in a dataset. In this study, we extend an ensemble technique that accurately predicts class labels and has the advantage of indicating which parameter constraints are most useful for predicting those labels. We develop a method for using this ranking to select features and remove less predictive attributes. We illustrate our method on a dataset containing images of potential supernovas, where we select 21 out of 39 features without reducing classification accuracy. We also extend the rule ensemble method to multi-class classification and compare it with the boosting and bagging ensemble methods on various classical multi-class datasets.